PROPERTIES OF FIXED POINT SETS AND A 
CHARACTERIZATION OF THE BALL IN C n 



BUM A L. FRIDMAN AND DAOWEI MA 

Abstract. We study the fixed point sets of holomorphic self- 
maps of a bounded domain in C™. Specifically we investigate the 
least number of fixed points in general position in the domain that 
forces any automorphism (or endomorphism) to be the identity. 
We have discovered that in terms of this number one can give the 
necessary and sufficient condition for the domain to be biholomor- 
phic to the unit ball. Other theorems and examples generalize and 
complete previous results in this area, especially the recent work 
of Jean-Pierre Vigue. 



0. Introduction 

Let D C C n be a bounded domain. Below we consider two families 
of self-maps of D. The first is the group Aut(D) of holomorphic au- 
tomorphisms of D; the second is the set H(D, D) of all holomorphic 
maps from D to D, i.e., the set of endomorphisms of D. 

Definition 0.1. A set K C D is called a determining subset of D with 
respect to Aut(D) (or H(D, D) resp.) if, whenever g is automorphism 
(resp. endomorphism) of D such that g(k) = k Vk G K , then g is the 
identity map of D. 

The notion of a determining set was introduced earlier in a pa- 
per we wrote with our collaborators Steven G. Krantz and Kang- 
Tae Kim [FK1]. In that paper we attempted to find a higher di- 
mensional analog of the following result of classical function theory 
([FF],[ES],[Mas],[PL],[Su]): if / : M -> M is a conformal self-mapping 
of a plane domain M which fixes three distinct points then /(£) = (. 
Determining sets have been further investigated in the following papers 
[FK2], [KK], [Vil], [Vi2]. 

Let W S {D) denote the set of s-tuples (xi, . . . , x s ), where Xj G D, 
such that {x%, . . . ,x s } is a determining set with respect to Aut(D). 
Similarly, W S (D) denotes the set of s-tuples (xi, . . . ,x s ) such that 
{xi, . . . , x s } is a determining set with respect to H(D, D). So W S (D) C 
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W S (D) C D s . We now introduce two numbers sq{D) and 'so(D). In 
case Aut(D) = id, s (D) = 0, otherwise s (D) is the least integer s, 
such that W S (D) ^ 0. The symbol 'sq(D) denotes the least integer s 
such that W S (D) ^ 0. Hence, s (D) < s (D). 

In [FK1] we proved the inequality s (D) < n + 1 for many (but not 
all) bounded domains in C n . In [Vil] J.-P. Vigue, using a different 
method proved this estimate for all bounded domains in C n . Further- 
more in [Vi2] Vigue proved the estimate %(D) < n+ 1 for all bounded 
domains in C n . 

Both estimates are the best possible, since for the unit ball B n C C n , 
s (B n ) — n + 1. In section 1 we prove that the reverse is true: if 
Sq(D) =n + l for a bounded domain D C C n , then D is biholomorphic 
to the unit ball 5™. Obviously, So(-D) depends on how large Aut(D) 
is: for a smaller group, we expect a lower So(-D)- This relationship is 
reflected below in Corollary 1.7. 

If a positive integer s > s (-D), then W S (D) ^ 0, so there are s 
points such that if an automorphism of D fixes these points it will 
fix any point of D. Now the question arises whether the choice of 
these s points is generic. To make it more precise we need to find 
out if W S (D) is open and everywhere dense in D s . We consider this 
question in section 2. Refining and complementing the results of [FK1, 
Vil, Vi2] we prove that W S (D) is open, and also dense if not empty. 
For the similar question related to W S (D) we provide examples to the 
contrary. 

1. Estimates for s (D) and a characterization of the ball 

IN C n 

1.1. Characterization of the ball by determining sets. This sec- 
tion is devoted to the proof of the following theorem. 

Theorem 1.1. Let D be a bounded domain in C n . Then sq(D) — n + 1 
if and only if D is biholomorphic to the unit ball B n in C n . 

To verify the estimate So(B n ) = n + 1 we need to prove that no 
n points in B n form a determining set for Aut(B n ). (This was done 
in [FK2], we repeat it here for completeness). Consider n arbitrary 
points (po,Pi, ■■■,Pn-i), where Pi G B n for i = 0, ...,n — 1. Consider 
g G Aut(B n ) such that g(po) = 0. Consider now n — 1 vectors 
g(pi), and the complex linear space 7r spanned by these vectors. Since 
dim(7r) < n — 1, there is a rotation / G Aut(B n ) that is not the 
identity and keeps all the points of n fixed. Now the automorphism 
h = g^ 1 o f o g g Aut(B n ) is not the identity and it fixes all n points 
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(po,Pi, ■■■iPn-i)- We proved that W n (B n ) = 0, so s (B n ) — n + 1. 



The rest of this section will be devoted to the proof that s (D) = n + 1 
implies that D is biholomorphic to the unit ball. 

If H is (isomorphic to) a subgroup of the unitary group U(n), let 
k(H) denote the least number k of vectors u±, . . . ,Uk such that if h G H 
and if h(uj) = Uj for j = l,...,k then h = id. For z G D the 
isotropy group Aut z (D) is isomorphic to the group of its differentials 
at z, and these differentials are unitary with respect to the Bergman 
inner product on the tangent space T Z (D). So Aut z (D) is isomorphic 
to a subgroup of U(n). 

Lemma 1.2. For a bounded domain D inC", s (D) < l+mm{k(Aut x (D)) : 
x G D}. 

Proof. Choose z G D so that k(Aut z (D)) = mm{k(Aut x (D)) : x G D}. 
Denote that number by k. Let ui, . . . , Uk be vectors in T Z D such that 
if h G Aut z (D) and if dh(z)(uj) = Uj for j = 1, . . . , k then dh = id 
(hence h = id). For each Uj, let Zj be a point on the geodesic through z 
in the direction Uj, so close to z that the geodesic is the unique length 
minimizing geodesic from z to Zj. Let / be an automorphism of D fixing 
z, Zi, . . . , Zk- Then df(z) fixes u±, . . . , u^. It follows that df(z) = id and 
/ = id. Therefore, s {D) < 1 + mm{k(Aut x (D)) : x G D}. □ 

Lemma 1.3. If H is a subgroup ofU(2) and if H is not transitive on 
S 3 , then k(H) < I. 

Proof. Let S = S 3 be the unit sphere in C 2 . It suffices to show that 
the set of fixed points in S of nontrivial elements of H (that is, each 
of these points is a fixed point of at least one nontrivial element of H) 
is not equal to S. For g,h G U (2) and x G S, x is a fixed point of h 
iff g~ x x is a fixed point of g~ l hg. So, without any loss of generality we 
can replace H with a subgroup of U{2) conjugate to H. 

The Lie algebra Q of U{2) consists of skew Hermitian matrices, so it 
has as a basis the following elements: 

a= (o o)' b= (o <)' c= (! "o 1 )' d= (i o)- 

Their Lie bracket relations are 

[a, 6] = 0, [a, c] = —d, [a, d] = c, [b, c] = d, [6, d] = — c, [c, d] = —2a + 26. 

If dimiJ = 4, then H — U(2), which contradicts the hypothesis that 
H is not transitive on S. 
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Suppose that dimH = 3. One can verify that the only 3-dimensional 
Lie subalgebra of Q is spanned by {c,d,a — b}. Hence, the identity 
component H of H is SU(2), again contradicting the hypothesis that 
H is not transitive on S. 

Now suppose that dimH = 2. One can verify directly that U{2) 
does not have a subgroup of dimension 2 and rank 1. Thus, H has 
rank 2. Up to conjugation, the identity component H$ of H is T 2 = 
{diag(e ia ,e il3 ) : a,f3 e R}. Each component of H is gT 2 . If h e gT 2 
has a nonzero fixed vector, h must satisfy det(h — id) = 0. It follows 
that if dimH < 2 then the set U of the nontrivial elements of H that 
have a fixed point on 5 has dimension < 1, and for each g <E U, the 
set of fixed points of g on S has dimension 1. Thus the set P of fixed 
points of nontrivial elements of H has dimension < 2. It follows that 
P^S. Therefore, k(H) < 1. 

□ 

Lemma 1.4. If H is a subgroup of U{n) with n > 2 and if H is not 

transitive on S 211 ^ 1 then k(H) <n — l. 

Proof. The case where n = 2 is the previous lemma. Suppose n > 2 
and H C U(n) is not transitive on S 2 ™' 1 . Choose x,y G so that 

no element of H maps x to y. Choose z G S* 2 ™ -1 orthogonal to both 
x and y. Let Si = G S 271 ^ 1 : (v,z) = 0}, where (v,z) = Yl v j~Zj, 
and let Hi — {g G H : g{z) = z}, and U\ — {g G £/(n) : ^(z) = z}. 
Now U\ = U{n — 1), and i^i is a subgroup of U\. By the induction 
hypothesis, since Hi is not transitive on Si, Hi has a determining set 
of n — 2 vectors {u>i, . . . , u>„_2}- It follows that {z, Wi, . . . , ti) n „ 2 } is a 
determining set for H. Therefore, k(H) < n — 1. □ 

The proof of our main theorem follows from the following 

Lemma 1.5. If D is a bounded domain in C n , and D ^ B n , then 
s (D) < n. 

Proof. If n — 1 the statement is obviously true. Assume n > 2. Let 
z E D. Since D ^ B n , Aut z (D) is not transitive on the directions at 
z, by the main result of [GK]. By the Lemma 1.4, k(Aut z (D)) < n — 1. 
It follows that s (D) < 1 + k(Aut z (D)) < n. □ 

Remark. For endomorphisms H(D, D) we still have for the ball 'so(B n ) = 
n + 1. However in the case of endomorphisms there are domains not 
biholomorphic to the ball but with the same maximum possible value 
of so- Here are two examples. 

For n = l, s'o(D) = s'oiB 1 ) = 2 for any bounded domain D C C. 
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We will show now that for n = 2, for the unit polydisc A 2 , so(A 2 ) = 
MB 2 ) = 3. 

Indeed, consider any two distinct points Pi,P2 G A 2 . Since the 
Aut( A 2 ) is transitive, we can find an automorphism g, such that g(pi) = 

0. Let L be the complex line through the origin and g{p2)- In terms of 
the coordinates this line can always be described in one of the forms: 
Z2 = Xzi, or z\ = Xz2 where |A| < 1. One can check that the map 
P : (zi,z 2 ) — > (zi,Xzi) in the first case, or P : (zi,z 2 ) — > (Xz 2 ,Z2) in 
the second case will produce a holomorphic retraction of the polydisc, 
fixing g(pi), gij>2)- Now the map g~ x Pg is a holomorphic retraction of 
A 2 fixing pi,P2- Therefore So(A 2 ) > 2. Since this number is also < 3, 
we conclude so(A 2 ) =3. 

1.2. An estimate for s (D). Let G be a subgroup of Aut(D). By 
So(-D, G) we denote the minimum number of distinct points in D such 
that if g G G, and g fixes all these points, then g = id. So, so(D) = 
s (D,Aut(D)). 

Theorem 1.6. Let D be a bounded domain in C n , let G be a subgroup 
of Aut(D), and let q = dimG. If q > 1, then s (D,G) < q. Ifq — 0, 
then s (D,G) < I. 

Proof. First we consider the case where q < 1. Let e denote the identity 
element of G, and let Q = G\{e}. For each g e Q, the set {x e D : 
g(x) = x} is an analytic set of D of dimension < In — 2. The set 
Wi := {(<?,£) E Q x D : g(x) = x} is an analytic set of Q x D of 
dimension < (2n — 2)+g < In — 1 < dimD. Let VF denote the set 
of fixed points of nontrivial elements of G. Since W = ir(Wi), where 
7r : Q x D — > D is the projection, and since dimW\ < dimD, we see 
that ^ £>. Therefore, s ( J D, G) < 1. 

Now we assume that q > 2. There must be an orbit Q of G of 
positive dimension. Let x G Q, and let if := G x be the subgroup of G 
consisting of elements g satisfying g(x) = x. Then dimiJ < dimC By 
induction hypothesis, so(D,H) < dimG — 1. Therefore, so(D,G) < 
1 + s (D, H) < dimG. □ 

Corollary 1.7. Lei D be a bounded domain in C n . If dim( Aut(D)) > 

1, then s (D) < dim(Aut(D)). If dim( Aut(D)) = 0, then s (D) < I. 

2. On topological properties of determining sets. 

2.1. Determining sets W S (D) are open and dense. Our aim in 

this section is to prove the following theorem. 

5 



Theorem 2.1. Let D be a bounded domain in C n and s > 1. Then 
W S (D) C D s is open; if in addition W S (D) ^ ; then W S (D) is dense 
in D s . 

The assertion that W S (D) is open and dense in D s was proved for 
some domains and s > n + 1 in [FK1]. Using analytic methods of 
[Cal],[Ca2], J.P. Vigue (see [Vil],[Vi2]) proved that W S (D) is open for 
all bounded domains and all s, and that it is dense for s > n + 1. 
By using the Bergman metric on a bounded domain we are able to 
use differential geometry methods and the Lie group properties (see 
[GKM], [Kl], [MZ], also [BD],[FMP], [Ma]) to prove the above general 
theorem. 

First we introduce some notation. If G is a subgroup of Aut(D), 
W s> c{D) denotes the set of s-tuples (x±, . . . ,x s ), where Xj G D, such 
that each element g G G satisfying g(xj) = Xj for j = 1, . . . , s has to 
be the identity. 

We need the following lemma (Theorem 2.4 in [Ma]). 

Lemma 2.2. Let Q be a bounded domain in C n containing the closure 
of the unit ball, and G a compact Lie subgroup of Aut{VL). Suppose that 
each G -orbit lies in a ball of radius 1/2. Then G = {id}. 

Lemma 2.3. Suppose that D is a bounded domain in C n and G is a 
subgroup of Aut(D). Then Wi t c{D) is open in D. 

Proof. We need to consider only the case W^g{D) ^ . Suppose that 
x G Wi t a(D), and there is a sequence of points in D, Xk — > x such 
that x k ^ Wi : g(D) \/k. Let U be a neighborhood of x, U C D, then 
there is a positive r such that for large enough k the ball with the 
center in x k and of radius r, b(xj,,r) in the Bergman metric compactly 
lies in U. The assertion that x^ ^ Wi t o(D) means that the subgroup 
G Xk of G fixing Xk is not the identity. This subgroup is a compact Lie 
subgroup of G, and also acts on b(xk,r) (since Bergman metric is an 
invariant metric). Applying (a properly adjusted form of) Lemma 2.2, 
one concludes that there exists an e > 0, such that for large enough 
k one has an automorphism g k G G Xk and a point y k G b(x k ,r) C U 
such that the Eucledian distance \g k {y k ) — y k \ > s. One can now find 
a subsequence {kj} such that (1) y kj — > y G U, and (2) g k — > g G G. 
We conclude now that g(x) = x and that \g(y) — y\ > e. This means 
that x W lt a{D) which is a contradiction. Therefore Wifi(D) is open 
in D. □ 

Let p(-, •) denote the Bergman distance. Let b(z, r) denote the Bergman 
ball with center z and radius r. Let b(z,r) be the closure of b(z,r) in 
D. 
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Lemma 2.4. Suppose that D is a bounded domain in C n and G is a 
subgroup of Aut(D). IfW ltG (D) ^ then Wi )G (D) is dense in D. 

Proof. In this proof, let W = Wi t a(D). Suppose that W is not dense 
in D. Then the closure K of W in D is not equal to D. Let p 
be a boundary point of K in D. Choose r > such that the clo- 
sure of b(p, 4r) in D is compact and such that each pair of points of 
b(p, 4r) is connected by a unique length-minimizing geodesic segment 
(in the Bergman metric). There exist points z,w such that p(z,p) < r, 
p(w,p) < r, w G W, and z £ K. Note that the orbit of w, G(w) C W. 
Let Q = G(w) C\b(p, 4r). Then Q is compact and Q C W. Let u be 
a point of Q nearest to z. Then -u is also a point of G(w) nearest to 
z, and R := p(z,u) < p(z,w) < 2r. Choose a point y on the unique 
length-minimizing geodesic segment from z to u such that y ^ K and 
y ^ z. For each point x of G(w), we see that 

and that the two equalities hold simultaneously only if x — u. Hence, 
p(z,y)+p(y,x) > p(z,u) = R for each x € G(w) , x ^ u. It follows that 
p(y,x) > R — p(z,y) = p(y,u) for each x G G(w), x ^ u. Therefore, 
u is the unique point of G(w) nearest to y. Since y K, there is a 
nontrivial g G G such that = y. Now p(y,u) = p(g(y), g(u)) = 
p(y,g(u)) forces g(u) = u. Since u G W, the map ^ must be the 
identity, contradicting the fact that g is not trivial. Therefore, Wi t c(D) 
is dense in D. □ 

Proof of Theorem 2.1. 

We need to prove this theorem only for W S (.D) 7^ 0. For 5 G Aut(D) 
let (^(g) denote the mapping 

Q s (#) :D S ^D S , Q s (g)( Zl ,...,z s ) = (g( Zl ), . . . , g(z s )). 

Let G = {Qs(g) ■ g e Aut(D)}. Then G C Awt(D s ), and W 1;G (D S ) = 
W S (D). By the previous lemmas, W S (D) is open and dense in D s . 

2.2. Determining sets W S (D) that are not open. In [Vi2] it was 
proved that W S (D) is open in D s for any bounded taut domain in C n . 
Our aim in this section is to present an example in C 2 of a bounded 
domain such that the determining set W^(-D) is not open in D 2 . 
First we construct the set D C C 2 . 

Denote B 2 = {z G C 2 ||z| < 2},B 1 = {z G C 2 ||^| < 1},^ = {z = 
(z 1 ,z 2 ) G C 2 !!^! 2 + \z 2 + 1| 2 < (1 - lO- 4 ) 2 }^ = {z = (z u z 2 ) G 
C 2 ||^i - 0.02| 2 + \z 2 + 1| 2 < 1}. Let Q, = (B 2 \B 1 ) U 61 U (6 2 n S 2 ). 
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Consider now pairs of points Pj,qj E Q , Pj = (—1.5,2 i),qj = 
(1.5, 2~ j ), and bydiscs Uj = {z = (z 1 ,z 2 ) G C 2 ||^i| < 1.5, \z 2 - 2~ j \ < 

oo 

2- 2J }; j = 1,2, .... And, finally domain D = Q U \J Uj. 

j'=i 

Note the following properties: 

1. D is a connected domain, D C B 2 . 

2. The entire complex disc Aj = {2; = (zi, z 2 ) G B 2 \z 2 = 2~ j } C D 
for all j. 

3. limj^ooPj = Po = (—1-5, 0), lim^oo qj = q = (1.5, 0); and for the 
disc A = {z = {z 1 ,z 2 ) G B 2 \z 2 = 0}, A n D = {z e C 2 1 1 < |^| < 
2,22 = 0). 

4. Ui nUj = for i ^ j for large enough 

5. Let Zq — (0, — 10~ 4 ) G 61 be the "tip" of this ball, and the point 
of bi closest to the origin. Let d denote the Kobayashi distance in B 2 
from the origin to Iq, k = k(0, d) the closed Kobayashi ball with the 
center at the origin and radius d. Then k D (61 U b 2 ) = {/ }- 

We are going to show that for any j the pair (pj, qj) ^ W 2 (D), but 

their limit (po,<Zo) G W 2 (D), which will prove that W 2 (D) is not open 
in/) 2 . 

Statement 1. For any j, there is a holomorphic retraction of -B2 
(and therefore of D C B 2 ) onto A 3 -, and since (pj,qj) G Aj, the pair 
( Pj , qj )<£W 2 (D). 

To prove this one needs to use first an automorphism g of B 2 to move 
Aj to Ao, use the natural projection P of the ball B 2 onto Ao, and set 
the needed holomorphic retraction &s g^ 1 o P o g. 

Statement 2. Any holomorphic map / : D — > D extends to a holo- 
morphic map F : B 2 ^ B 2 . 

Statement 3. Any holomorphic map F : _B 2 — > B 2 that fixes our two 
points po, qo will fix all the points of A . 

For proof see ([Vi2], ex. 1 in sec 4). 

Now let / : D — > D be a holomorphic map that fixes our two points 
Po, q - Its extension F : B 2 ^ B 2 will be an identity on A . 
Statement 4. F(Z ) = V 

Indeed, consider K the Kobayashi ball in B 2 with center at the origin 
that coincides with the standard unit ball in C 2 . K n D consists or 
nonintersecting connected pieces, only one of which, namely G — K n 
(61 U62) has a point (0.02, 0) on Ao as a limit point. Since this point is 
fixed by F, and F cannot increase the Kobayashi distance, we conclude 
that F(G) C G. 



8 



Since it is also true that F( k) C k, the only possible image for l 
under F is the point itself (see property 5 above), so F( l ) = l . 

We now conclude our observation by pointing out that F : B 2 — > B 2 
has three fixed points po,qo,lo- By ([Vi2], ex. 1 in sec 4) F is the 
identity, so / : D — > -D is the identity, and therefore (po, ?o) £ W 2 {D). 

2.3. Determining sets W S (.D) that are not dense. Our goal here 
is to present an example of a domain D such that for any s the deter- 
mining set W S (D) is not dense in D s . 

J. -P. Vigue (see [Vi2], ex. 2 in sec 4) has provided such an example 
for D = A 2 = {z = (z 1 ,z 2 ) e C 2 ||zi| < 1, \z 2 \ < 1} the polydisc in C 2 , 
and s = 3. For completeness, using the same idea, we provide here an 
example for A 2 and any s > 3. 

Fix s points (A,-, 0), Aj = 2~ J , j = 1, s. The set T of these points is 
a point in (A 2 ) 5 . T ^ W^A 2 ), since (zi,z 2 ) i— > (21, 0) is a holomorphic 
retraction of A 2 , fixing all these points. Let T = {(%, fej), j = 1, s} G 

(A 2 ) 6 ' be any 5 > perturbation of T. So, f2(\ A j ~ %| 2 + l&il 2 ) < 

We will show that if 5 is small enough T ^ ^(A 2 ). 

Consider the Lagrange interpolation polynomial y?(u>) = X] EI 7^~r 

One can verify that if 5 > is small enough (say 5 < 4~ s ) then 
|(/9(w)| < 1 if |iu| < 1. Now the map (zi,z 2 ) > (zi,</?(zi)) is a holomor- 
phic retraction of the unit polydisc that has T in its set of fixed points. 
Therefore f £ W S {A 2 ). 

As a remark we note that using this idea one can construct many 
such examples. Moreover the following theorem holds. 

Theorem 2.5. Consider the topological space D n of all bounded do- 
mains in C n with the topology induced by the Hausdorff distance be- 
tween boundaries of the domains. Let G n C D n be such that if D e G n 
then W S (D) is not dense in D s for all s > 1. Then G n is dense in the 
topological space D n . 
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